Search for: All records

Despite the empirical success of foundation models, we do not have a systematic characterization of the representations that these models learn. In this paper, we establish the contexture theory. It shows that a large class of representation learning methods can be characterized as learning from the association between the input and a context variable. Specifically, we show that many popular methods aim to approximate the top-d singular functions of the expectation operator induced by the context, in which case we say that the representation learns the contexture. We demonstrate the generality of the contexture theory by proving that representation learning within various learning paradigms—supervised, self-supervised, and manifold learning—can all be studied from such a perspective. We also prove that the representations that learn the contexture are optimal on those tasks that are compatible with the context. One important implication of the contexture theory is that once the model is large enough to approximate the top singular functions, further scaling up the model size yields diminishing returns. Therefore, scaling is not all we need, and further improvement requires better contexts. To this end, we study how to evaluate the usefulness of a context without knowing the downstream tasks. We propose a metric and show by experiments that it correlates well with the actual performance of the encoder on many real datasets. 

We consider the task of heavy-tailed statistical estimation given streaming p-dimensional samples. This could also be viewed as stochastic optimization under heavy-tailed distributions, with an additional O(p) space complexity constraint. We design a clipped stochastic gradient descent algorithm and provide an improved analysis, under a more nuanced condition on the noise of the stochastic gradients, which we show is critical when analyzing stochastic optimization problems arising from general statistical estimation problems. Our results guarantee convergence not just in expectation but with exponential concentration, and moreover does so using O(1) batch size. We provide consequences of our results for mean estimation and linear regression. Finally, we provide empirical corroboration of our results and algorithms via synthetic experiments for mean estimation and linear regression. 

Abstract Magnonicsis a research field that has gained an increasing interest in both the fundamental and applied sciences in recent years. This field aims to explore and functionalize collective spin excitations in magnetically ordered materials for modern information technologies, sensing applications and advanced computational schemes. Spin waves, also known as magnons, carry spin angular momenta that allow for the transmission, storage and processing of information without moving charges. In integrated circuits, magnons enable on-chip data processing at ultrahigh frequencies without the Joule heating, which currently limits clock frequencies in conventional data processors to a few GHz. Recent developments in the field indicate that functional magnonic building blocks for in-memory computation, neural networks and Ising machines are within reach. At the same time, the miniaturization of magnonic circuits advances continuously as the synergy of materials science, electrical engineering and nanotechnology allows for novel on-chip excitation and detection schemes. Such circuits can already enable magnon wavelengths of 50 nm at microwave frequencies in a 5G frequency band. Research into non-charge-based technologies is urgently needed in view of the rapid growth of machine learning and artificial intelligence applications, which consume substantial energy when implemented on conventional data processing units. In its first part, the 2024 Magnonics Roadmap provides an update on the recent developments and achievements in the field of nano-magnonics while defining its future avenues and challenges. In its second part, the Roadmap addresses the rapidly growing research endeavors on hybrid structures and magnonics-enabled quantum engineering. We anticipate that these directions will continue to attract researchers to the field and, in addition to showcasing intriguing science, will enable unprecedented functionalities that enhance the efficiency of alternative information technologies and computational schemes.